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QUADRATIC INTERNAL MODEL PRINCIPLE IN 
MATHEMATICAL PROGRAMMING 


Neculai ANDREI 


Rezumat. Demonstram ca pentru.problemele de programare matematica continue un 
algoritm de optimizare trebuie sa incapsuleze intr-o maniera explicita sau implicita un 
model intern patratic.al problemei de rezolvat care reprezinta esenta problemei din punctul 
de vedere al algoritmului de optimizare. Modelele de optimizare se scriu pe baza legilor de 
conservare. Pentru acele sisteme care verifica principiul minimei acfiuni, teorema Noether 
exprima echivalenta dintre legile de conservare si simetrii. Dar matematic simetriile sunt 
exprimate prin forme patratice. Deci, la baza oricarui model real de optimizare se afla o 
forma patratica. Principiul modelului intern pdatratic zice ca aceasta forma patratica 
modificata pentru a include principalele ingrediente ale algoritmului de optimizare 
reprezinta modelul intern patratic al algoritmului de optimizare. 


Abstract. We show that for mathematical programming problems an optimization 
algorithm must encapsulate implicitly or explicitly a quadratic internal model of the 
problem to be solved which represents the essence of the problem from the view point of the 
algorithm. Optimization models are coming from the conservation laws. For systems which 
obey the principle of least action the Noether’s theorem expresses the equivalence between 
the conservation laws and symmetries. But mathematically, symmetries are expressed by 
quadratic forms. Therefore, at the heart of every real optimization model is a quadratic 
form. The quadratic internal model principle says that this quadratic form modified in order 
to imbed the main ingredients of the optimization algorithm represents the quadratic 
internal model of the optimization algorithm. 
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1. Introduction 


Starting with an initial point x, every algorithm for solving the general 
continuous nonlinear optimization problem 
min f(x) 
subject to (1) 
h(x) =0, 


where f:R"—R and h:R" —+R", can be considered as a generator of a 


sequence of points {x,} which satisfy the constraints of the problem in such a 
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way that f(x,)> f (x), where x’ is a local solution of the problem. The line 


search methods are characterized by two main actions. At the iteration k a search 
direction d, is generated and then a suitable point x,+a,d, is computed by a 
step length q@, so that a reduction of the minimizing function or of a merit 
function (a penalty function) is obtained. The main action in any optimization 
algorithm is the design of the generator of directions d,. The steplength is 


computed using the standard procedures of Armijo or of Wolfe in order to reduce 
the values of the function f or of a merit function. Plenty of nonlinear 


optimization algorithms are known and there are a lot of papers and books 
presenting them from the viewpoint of theoretical and computational aspects. 


To solve problem (1), or a more general version of it with inequality 
constraints, each optimization algorithm must “understand” it. There is a large 
diversity of optimization algorithms. Many of them solve a constrained 
optimization problem by converting it to a sequence of unconstrained problems 
via Lagrangian multipliers or via penalty or barrier functions. Another class of 
methods solves nonlinear programming problems by moving from a feasible point 
to a new improved one along a feasible direction. However, every optimization 
algorithm, in one way or another, is based on the Karush-Kuhn-Tucker optimality 
conditions. Generally, these conditions are expressed as a nonlinear algebraic 
system. In the framework of the Newton machine this nonlinear system is reduced 
to a sequence of linear algebraic systems, which is equivalent to a sequence of 
quadratic programming problems. The quadratic internal model principle in 
mathematical programming states that “an optimization algorithm must 
encapsulate implicitly or explicitly a quadratic internal model of the problem to 
be solved”. Every optimization algorithm uses its own quadratic internal model 
which takes into account the main ingredients defining the algorithm. This is the 
minimal part that must be encapsulated by the algorithm in order to solve the 
problem. 


The philosophical motivation behind the quadratic internal model principle in 
mathematical programming is as follows. As known, the mathematical model of a 
physical reality is based on the conservation laws. In physics, a conservation law 
states that a particular measurable property of an isolated system does not change 
while the system evolves. Any particular conservation law is a mathematical 
identity to certain symmetry of a physical system. For systems which obey the 
principle of the least action and therefore have a Lagrangian [Lagrange, 1797], 
[Fourier, 1798] the Noether’s theorem [Noether, 1971] expresses the equivalence 
between conservation laws and the invariance of physical laws with respect to 
certain transformations called symmetries. The behavior of a physical system can 
often be expressed in terms of a specific function of the system variables, called 
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Lagrangian. The system follows a path through the phase space such that the 
integral of the Lagrangian is stationary. For a system with Lagrangian Lof the 
variables g and g=dq/dt the equation of motion is 


d (=) _@L 

dt\ aq) aq’ 
From this equation Noether specified that if the quantity on the right hand term is 
zero (meaning that L is symmetrical overg), then the rate of change of the 
quantity in parentheses on the left side is also zero, i.e. it is a conserved quantity. 
Generally, any symmetry of the Lagrangian function corresponds to a conserved 


quantity, and vice versa. It seems that at the fundamentals of our cognoscible 
universe lie the concept of symmetry. 


But, mathematically symmetries are expressed by quadratic forms — a 
homogeneous polynomial of degree two in a number of variables. It is worth 
saying that the quadratic forms are central objects in mathematics and they are 
ubiquitous in physics and chemistry. Quadratic forms occur in number theory, 
Riemannian geometry, Lie theory and they always express energy of a system, 
particularly in relation to the L’ norm, which leads us to the use of the concept of 
Hilbert spaces. Therefore, it is quite natural to see that at the heart of every 
mathematical model is a quadratic form. This quadratic form must be replicated in 
an optimization algorithm in order to get a solution of the corresponding problem. 


2. Quadratic Internal Model Principle. 


In his synthesis, Yuan [2003] has shown that using the Newton machine for every 
method for solving a constrained optimization problem a linear algebraic system 
can be associated in a most natural way. All the linear systems corresponding to 
different methods are similar in form and can be expressed as: 


W(x,,.A*)+T, —Vh(x,) payee Vi (%,) he ®) 
VA(x,) ss n| | h(a)tee? 


where W(x,,4°)=V"f (%,)— ATW, (%,) is the Hessian matrix of the Lagrange 


i=l 
function L(x,/) = f (x)—Ah(x), T, ¢ R”” is a symmetric matrix, S$, ¢R”” isa 
null or a diagonal matrix whose elements are nonpositive, ¢,¢R" and ¢, € R” 
are two vectors. In (2) d is the searching direction and 77 is an auxiliary vector 


which for some methods could be the Lagrange multiplier. In the following we 
shall consider two cases. 
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1) Let us assume that S$, =0. Therefore, from (2) we get 


ie -Vh(x,)" | i“ _ sie é,| 


(3) 
VA(x,) 0 {L7 A(X) + & | 


It is easy to see that the system (3) corresponds to the Newton method for 
optimization with equality nonlinear constraints or to the sequential quadratic 
programming method. The augmented system (3) can be considered as the 
necessary condition for d to be a solution of the following quadratic 
programming problem 


min d"(W(x,,2') +7, )d —(Wf(4)+ 6)" 


subject to (4) 
V(x, + Gees, ) = 0. 


It is well known that if (W(x,,A")+T,) is positive definite on the null space of 
Vh(x,) and Vh(x,) is a full-rank matrix, then the quadratic problem (4) has a 


unique global solution d. This solution can be obtained by solving the augmented 
system (3), where d is the solution of the problem and 77 is the Lagrange 
multiplier associated to the equality constraint. The problem (4) is the quadratic 
internal model of problem (1) associated to the methods involving the linear 
system (3) (with S, =0). 


For example, the quadratic internal model of the problem (1) corresponding to 
the Newton method 1s: 


min d"W(y,2" )d —(Vf(x,)+ Vax, A‘) d 


subject to (5) 
Vi(x,)d + h(x,) =0. 


2) Let us suppose that S, #0. Normally, S, is a diagonal matrix, whose diagonal 
elements are all negative. In this case it is easy to see that the system (2) 
corresponds to the following methods: the augmented Lagrange function, the 
inverse barrier function, the log-barrier function, the interior point algorithms, the 
path-following methods, the affine scaling interior point methods etc. From (2) we 
get 


1 =—S,'Vh(x,)d —Soh(x,)-S¢le,. (6) 
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Therefore, using (6) in (2) it follows that 


[Wy AN +T, +VA(x,)" S,'VhCx,) |d 
=—Vh(x,)' S'h(x,) -Vh(x,)' S'&) — Vf (4%) -&- (7) 


But, (7) is equivalent with the following quadratic problem 


mind" [WO AN) +7, +VACx,)! SeVhCx,) Jd 


= [Wh (x)! 8,0) = VAC) S,'e — VE (a) -& | (8) 


which is called the quadratic internal model of problem (1) associated to the 
methods involving the linear system (2) (with S, #0). 


Therefore, an optimization algorithm for solving (1) must encapsulate a 
procedure for solving (in an iterative way) the quadratic internal model (8), 
which represents the essence of the problem from the view point of the algorithm 
involving (2). 


Observe that the quadratic internal model of (1), as expressed by (8), is dependent 
on the algorithm we consider for solving the problem (1). In particular, for 
example, the quadratic internal model of problem (1) corresponding to the 
augmented Lagrange function method, in which the augmented Lagrange function 
is 


i” (9) 


L(x,A,0) = f (x) +A7h(x) + +o a(x) 
where 4 € R” is the Lagrange multiplier and o > 0 is the penalty parameter, is: 
min ; d' (W(x,,A‘h(x,)) + OVA(x,)' VA(x,))d — 
(-Vf (a) +VA(x,)' (AS —oh(x,)))d. (10) 


In this case 7 € R” is given by 7 =o0Vh(x, )d+oh(x,). 


It is worth saying that for the unconstrained problem (m=0) the Newton step can 
be obtained by solving the following quadratic problem 


min d"V" f(,)d +Vf (x,)' d (11) 


which is the quadratic internal model of the problem min f (x) corresponding to 
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the Newton method. Of course, the Newton step d =(V? Ff (x,)) ‘Vf (ay) 18 


obtained by solving the linear system (Vf (x,))d =—Vf(x,), but as we know, it 


comes from the quadratic problem (11). Similarly, we can say that for the 
unconstrained problem the quasi-Newton step can be obtained by solving the 
following quadratic problem 


mind" Bia +VE (ap) d (12) 


which is the quadratic internal model of the problem min f (x) corresponding to 
the quasi-Newton method, where B, is a positive definite matrix satisfying the 


quasi-Newton equation B, (x,,, —x,) = Vf (%,.,) —- VFO). 


Conclusion 


To solve a mathematical programming problem an algorithm must encapsulate in 
an implicitly or explicitly manner a quadratic internal model of the problem. This 
is the quadratic internal model principle in mathematical programming. This 
quadratic internal model reflects the ingredients of the algorithm and represents its 
essence. The philosophical support of this principle is coming from the Noether 
Theorem which expresses the equivalence between the conservation laws and 
symmetries which can be represented by quadratic forms. These quadratic forms 
are the fundamentals of every line search optimization algorithm. 
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